Air traffic operations are expected to increase significantly. 
Automation must maintain or exceed current safety standards 


Separation Assurance — algorithms and systems gradually taking the 
role of air-traffic controllers to enable reduced aircraft separation 


Onboard-Collision Avoidance Systems — TCAS, ACAS X 


A “Traffic, Traffic” annunciation —- 
indicates a potential maneuver Resolution advisories: 
may be required 


A ¢ Climb or descend 
¢ Level off 


e¢ Maintain climb or descend 


¢ Don't or limit climb or descent rate 


40 secs from Near-Mid-Air Collision (NMAC) 


state variables 
— h,,,: relative altitude, in [-1000...1000] ft 
— dh,,,, /dh,., : ownship / intruder climb 


own int ° 


rates, in [-2500...2500] ft/min 


Aprev / Sra: advisory issued by ACAS X in 
previous sec / current pilot state, both in 
{COC, CL/DES1500, SCL/SDES1500, SCL/SDES2500 


update and advisory frequency is set to 1 sec 


discretization resolution n for a variable V 
means that V is discretized to n points above 
and n points below O within its interval of 
values. For example, discretization resolution 
of 10 for h,,, means: 


{-1000, -900, -800, ... , 0, 100, ..., 900, 1000} 


intruder 


SN 
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int 
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own 


ownship 


CL 1500 ft/min COC 


P=04 P=0.6 


minimize costs / maximize rewards 


NMAC (near-mid-air collision 
Alert (from COC to advisory) 
Strengthening (strengthen advisory) 


Reversal (e.g. climb to descend) 
COC (clear of conflict) 


each grid-point in the look up table has corresponding costs for each advisory; 
ACAS X returns the advisory with the smallest cost. 


ControllerMenu 


ControllerMenu 


verification starts with asking questions 


did they pick the right costs? 


what if the pilot reacts late? 


..and proceeds with answering them 


DR, : model discretization resolution for look-up table generation; 
baseline [KC 2011] resolution is (dh,,,,=10, dh,,,=10, h,,,=10) 
DR, : model discretization resolution to model check look-up table 


continuous DR, (higher resolution 
ir model haa DR,) 
generation evaluation 
model model 
look-up table : 
model 


checking 


=" we compute P(NMAC) of the baseline look up table deployed in 
models that are obtained through discretization with varying 
resolutions DR, (dh,,,, dhj.+, A.) 


own? 


— ALL varies climb rates and relative altitude in DR,: (20, 20, 20), (30, 30, 30), ... 
— climb varies climb rates only in DR,: (20, 20, 10), (30, 30, 10), ... 
— alt varies relative altitude only in DR,: (10, 10, 20), (10, 10, 30), ... 


model checking resolution DR; 


=» P(NMAC) decreases with higher evaluation resolutions 
= relative altitude discretization is indicative 


allows precise qutomated analysis of probabilistic properties 


expressed in a formal logic such as PCTL; generates encounters that 
exhibit property-related behaviors 


= what is the probability of NWUAC? (P=?[F NMAC]) 2.5 x 107 
= what if pilot responds immediately? 
(P=?(F NMAC | Ga, ey = Spa)) 24% 10" 


= what is the probability of a split advisory? 1.8 x 10° 
P=?[F(!COC A P=1[X COC] A P>0 [F !COC] )] 

= split advisories are harder to directly take into account during look 
up table generation because they require to record history 
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time to LHS 


(reward for COC + cost of alert) < cost of reversal (“sneaky” reversals) 


synthesis / design 


= tune look-up tables based on minimum acceptable performance 


— deterministic look-up tables based on weights form a convex Pareto front; we 
implement algorithms that approximate it above target performance 
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DR, : model discretization resolution for look-up table generation; 
baseline [KC 2011] resolution is (dh.,.=10, dh,.,=10, h,.,=10) 


own int rel 


DR- : model discretization resolution to model check look-up table 


continuous DR, (higher resolution 


model fen DR,) 


evaluation 
model 
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model 
checking 


= question: what is the effect of resolution discretization DR, on look- 
up table synthesis? 


= experiment set up: 


— evaluate baseline (dh,,,,=10, dh,,,=10, h,.,=10) [KC 2011] in new DR, 


int 


— use result as target for synthesis 


= how we vary resolutions DR, (dh... Ahir, Ape) 
— ALL varies climb rates and relative altitude in DR¢: (20, 20, 20), (30, 30, 30), ... 
— climb varies climb rates only in DR: (20, 20, 10), (30, 30, 10), ... 
— alt varies relative altitude only in DR¢: (10, 10, 20), (10, 10, 30), ... 


— note that alt results in the smallest look up tables — in terms of numbers of 
states — for each value increase 


= compare the synthesized look-up tables in DR, = (50, 50, 100) 
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generation resolution DRg 


recommendation: (10, 10, 30), or (n, n, 3*n) 


we could not use off-the-shelf tools, so we built VeriCA toolset 
— our tools support models written in Java 
—- we customized verification and synthesis algorithms for ACAS X needs 


we analyzed ACAS X version that we reproduced based on: 


Kochenderfer, M. J., and Chryssanthacopoulos, J. P. Robust airborne collision avoidance 
through dynamic programming. Project Report ATC-371, Massachusetts Institute of 
Technology, Lincoln Laboratory, 2011. 


ETAPS 2014 EASST best paper award 


— Christian von Essen, Dimitra Giannakopoulou: Analyzing the Next Generation 
Airborne Collision Avoidance System, TACAS 2014. 


FAA / NASA Ames Interagency Agreement for ACAS X and VeriCA 
— thus able to apply our subsequent work on the actual ACAS X code 


model quality 
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State machine model with probabilistic transitions (MDP) is used to generate onboard 
look-up table. The MDP is not present in the onboard system. 
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State machine model with probabilistic transitions (MDP) is used to generate onboard 
look-up table. The MDP is not present in the onboard system. 


state estimate at time t 
on look-up table 


state estimate at time t+1 
t+1sec fae on look-up table 
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State machine model with probabilistic transitions (MDP) is used to generate onboard 
look-up table. The MDP is not present in the onboard system. 


MDP model 
state transitions 


t+1sec 


full conformance: 
ACAS X states contained 
in MDP states 
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State machine model with probabilistic transitions (MDP) is used to generate onboard 
look-up table. The MDP is not present in the onboard system. 


We defined Conformance Relations that compare MDP model to flight data 


MDP model 
state transitions 


ioe a 


full conformance: 
ACAS X states contained ACAS X and MDP states 
in MDP states do not intersect 


MDP model 
state transitions 


Data generation: Non-conforming encounters are rare in test data of 
the ACAS X distribution. We used a reinforcement learning framework 
to target generation of non-conforming simulated encounters. 


MDP 
Data Analysis: The intruder climb 


rate has been identified as a 0 
common factor for divergence 
across the data. Further analysis is 
needed. 
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Open question: Does non- 
conformance imply potential 
violation of safety requirements? 
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V&V of autonomy 


=" formulation of requirements is harder — autonomy-specific? 
— optimization, adaptive and learning algorithms 
— example: loss of separation, ACAS X 


no picked resolution is allowed to cause a more imminent secondary conflict 
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formulation of requirements is harder — autonomy-specific? 
— optimization, adaptive and learning algorithms 
— example: separation assurance, ACAS X 


need for advanced testing infrastructures 
— test case generation for stress-testing and requirements coverage 
— examples: ACAS X, separation assurance, autonomous vehicles 


V&V at runtime, including requirements 
— ACAS X (error prediction with statistical learning) 
— separation assurance 


trust 
— extensive verification 
— explanation of decision-making algorithms 
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